Image processing apparatus and control method thereof

ABSTRACT

An image processing apparatus that records a high dynamic range (HDR) signal as a file. The apparatus acquires information indicating peak luminance corresponding to one output dynamic range, of a plurality of different output dynamic ranges, in accordance with shooting settings of the HDR signal. The apparatus then records a first value based on the acquired information to the file along with the HDR signal.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of application Ser. No. 16/554,968,filed Aug. 29, 2019, the entire disclosure of which is herebyincorporated by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing apparatus and acontrol method thereof, and specifically relates to technology forhandling a high dynamic range (HDR) signal.

Description of the Related Art

Display apparatuses having a wider dynamic range of display luminancecompared with known apparatuses have been realized due to improvedperformance of light emitting elements (LEDs, for example) or the like.In such display apparatuses, an image (HDR image) having colors anddetails in a high luminance range that cannot be expressed in knowndisplay apparatuses can be faithfully displayed.

Signal characteristics illustrating the relationship between a videosignal level and display luminance in an HDR image are defined in EOTF(Electro-Optical Transfer Function). Also, there are two EOTFs, namelyPQ (Perceptual Quantization) standardized in SMPTE ST 2084 and HLG(Hybrid Log Gamma) standardized in ARIB STD-B67. A major differencebetween HLG and PQ is that the display luminance is handled as arelative value in HLG, but the display luminance is handled as anabsolute value whose maximum is 10000 nits (or cd/m²) in PQ. Therefore,when shooting is performed in a shooting mode in which the outputdynamic range changes, there may be a case where the display peakluminance changes in PQ.

FIG. 1 shows an example of the input-output characteristics when theoutput dynamic range changes between two shooting modes denoted by 11and 12, and the horizontal axis shows input steps [EV] and the verticalaxis shows output luminance. When the gamma curves of the two shootingmodes are compared, it is apparent that the input-output characteristicsare the same other than the high luminance range, and the peak luminancediffers as denoted by 13 and 14. Note that, hereinafter, the signalcharacteristics of an HDR image are assumed to be conformable to PQ,unless otherwise specifically noted.

Incidentally, an image signal (SDR signal) having a known displayluminance dynamic range (Standard Dynamic Range: SDR) is premised on aviewing environment with EOTF of γ=2.2 or the like, but an HDR imagesignal (HDR signal) is premised on a viewing environment with EOTF of PQor the like. Therefore, when an image is reproduced from an HDR signalin a viewing environment in which an SDR signal is envisioned, a videothat is different from the video intended by a producer is displayed. Inorder to avoid such a problem, for example, a tone mapping isconceivable in which the tone values of the HDR signal are converted(compressed) to tone values of the SDR signal for viewing in the SDRenvironment, although the effect of HDR is lost. Note that staticmetadata called maxCLL (maximum Content Light Level) indicating themaximum luminance of the contents can be added to the HDR signal. As aresult of referring to maxCLL, a reproduction device can specify themaximum luminance of the received HDR contents.

Japanese Patent Laid-Open No. 2017-184220 (Document 1) and JapanesePatent Laid-Open No. 2018-093530 (Document 2) disclose tone mappings forconverting an HDR signal to an SDR signal. Document 1 disclosesdesigning a tone mapping characteristic using a maximum luminance levelestimated using the shape of histogram of luminance values and maxCLL ofthe HDR signal. Also, Document 2 discloses designing a tone mappingcharacteristic in accordance with a geometric-formed region in an imagein order to suppress tone jump in a low frequency subject such as thesky.

In the method in Document 1, the mapping that maps the maximum luminancelevel of the HDR signal to the maximum value of the EOTF domain of SDRis performed, for example, even in a case where the HDR signal wasobtained by shooting in an underexposure condition. Therefore, themapped SDR signal viewed in the SDR environment may look brighter thanthe original HDR signal viewed in the HDR environment.

Also, in the method in Document 2, the tone mapping does not considerthe dynamic range of the HDR image and thus the output SDR signalsometimes may not represent the brightness intended by a producer.

SUMMARY OF THE INVENTION

The present invention has been made in view of the foregoing problem inthe known technologies. The present invention provides an imageprocessing apparatus and a control method thereof for converting an HDRsignal to a SDR signal such that the change in brightness can besuppressed between a case of viewing in an HDR environment and a case ofviewing in an SDR environment.

According to an aspect of the present invention, there is provided animage processing apparatus that records a high dynamic range (HDR)signal as a file, comprising: one or more processors that executeinstructions stored in a memory to function as: an acquisition unitconfigured to acquire information indicating peak luminancecorresponding to one output dynamic range, of a plurality of differentoutput dynamic ranges, in accordance with shooting settings of the HDRsignal; and a recording unit configured to record a first value based onthe information acquired by the acquisition unit to the file along withthe HDR signal.

According to another aspect of the present invention, there is providedan image processing apparatus that processes a high dynamic range (HDR)signal, comprising: one or more processors that execute instructionsstored in a memory to function as: an acquisition unit configured toacquire a first value indicating peak luminance in an output dynamicrange in accordance with shooting settings of the HDR signal, the firstvalue being recorded in association with the HDR signal, and aconversion unit configured to convert the HDR signal to a signal in apredetermined format by performing a tone-mapping that maps tone valuesof the HDR signal to tone values in the predetermined format having adifferent dynamic range, based on the first value.

According to a further aspect of the present invention, there isprovided a control method of an image processing apparatus that recordsa high dynamic range (HDR) signal as a file, the control methodcomprising: acquiring information indicating peak luminancecorresponding to one output dynamic range, of a plurality of differentoutput dynamic ranges, in accordance with shooting settings of the HDRsignal; and recording a first value based on the information acquired bythe acquiring to the file along with the HDR signal.

According to another aspect of the present invention, there is provideda control method of an image processing apparatus that processes a highdynamic range (HDR) signal, the control method comprising: acquiring afirst value indicating peak luminance in an output dynamic range inaccordance with shooting settings of the HDR signal, the first valuebeing recorded in association with the HDR signal, and converting theHDR signal to a signal in a predetermined format by performing atone-mapping that maps tone values of the HDR signal to tone values inthe predetermined format having a different dynamic range, based on thefirst value.

According to a further aspect of the present invention, there isprovided a non-transitory computer-readable medium that storesinstructions executable by a computer, wherein the instructions, whenexecuted by the computer, causes the computer to operate as an imageprocessing apparatus that records a high dynamic range (HDR) signal as afile, comprising: an acquisition unit configured to acquire informationindicating peak luminance corresponding to one output dynamic range, ofa plurality of different output dynamic ranges, in accordance withshooting settings of the HDR signal; and a recording unit configured torecord a first value based on the information acquired by theacquisition unit to the file along with the HDR signal.

According to another aspect of the present invention, there is provideda non-transitory computer-readable medium that stores instructionsexecutable by a computer, wherein the instructions, when executed by thecomputer, causes the computer to operate as an image processingapparatus that processes a high dynamic range (HDR) signal, comprising:an acquisition unit configured to acquire a first value indicating peakluminance in an output dynamic range in accordance with shootingsettings of the HDR signal, the first value being recorded inassociation with the HDR signal, and a conversion unit configured toconvert the HDR signal to a signal in a predetermined format byperforming a tone-mapping that maps tone values of the HDR signal totone values in the predetermined format having a different dynamicrange, based on the first value.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating gamma curves in shooting modes in whichthe peak luminance is different.

FIG. 2 is a cross sectional view of a general interchangeable lens typedigital single lens reflex camera.

FIG. 3 is a block diagram illustrating an exemplary configuration ofelectric circuit of a camera and an interchangeable lens.

FIG. 4 is a diagram illustrating a tone mapping operation in a firstembodiment.

FIG. 5 is a flowchart relating to the operation for calculating maxDRLin the first embodiment.

FIG. 6 is a diagram illustrating characteristics of PQ (PerceptualQuantization), which is EOTF standardized in SMPTE ST2084.

FIG. 7 is a diagram illustrating an example of a data file structureused in embodiments.

FIG. 8 is a flowchart relating to the tone mapping operation in thefirst embodiment.

FIG. 9 is a diagram illustrating an operation for updating maxDRL in asecond embodiment.

FIG. 10 is a flowchart relating to an operation for updating maxDRL inthe second embodiment.

FIGS. 11A to 11C are diagrams illustrating an example of retouching andan exemplary calculation of maxDRL_r in the second embodiment.

FIG. 12 is a flowchart relating to a tone mapping and an operation forupdating maxDRL in a third embodiment.

FIGS. 13A and 13B are diagrams illustrating an example of a data filestructure used in a fifth embodiment.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will now be described indetail in accordance with the accompanying drawings. Note that theembodiments to be described are merely illustrative, and do not limitthe scope of the present invention. For example, the following describesembodiments in which the present invention is applied to a digitalcamera. However, the digital camera is merely an example of an imageprocessing apparatus to which the present invention can be applied. Thepresent invention can be implemented in any electronic device. Such anelectronic device includes an image capture apparatus, of course, suchas a digital camera or a digital video camera, a personal computer, atablet terminal, a mobile phone, a game machine, a drive recorder, arobot, a drone, or the like, but there is no limitation thereto.

First Embodiment

FIG. 2 is a cross sectional view illustrating an exemplary arrangementof main optical members, sensors, and the like of a digital camera(hereinafter, referred to as a camera) according to the embodiments. Thecamera in the present embodiment is an interchangeable lens digitalsingle lens reflex camera, and includes a camera body 1 and aninterchangeable lens 2. In the camera body 1, an image sensor 10 is aCMOS image sensor or a CCD image sensor, for example, and in which aplurality of pixels (accumulation type photoelectric conversionelements) are arranged. A mechanical shutter 11 provided forward in thevicinity of the image sensor 10 controls the exposure timing andexposure period of the image sensor 10.

A semi-transmissive main mirror 3 and a first reflecting mirror 7 thatis arranged behind the main mirror 3 jump up when shooting is performed.A second reflecting mirror 8 reflects a light beam reflected by thefirst reflecting mirror 7 such that the reflected light beam is incidenton a focus detection sensor (AF sensor) 9. The AF sensor 9 may be animage sensor with fewer pixels than the image sensor 10, for example.

The first reflecting mirror 7, the second reflecting mirror 8, and thefocus detection sensor 9 are constituent elements for performing focusdetection using a phase difference detection method at any positioninside a shooting screen. A photometry sensor (AE sensor) 6 receiveslight of an image of the shooting screen formed by light reflected by apentaprism 4 and a third reflecting mirror 5. The AE sensor 6, in whicha light receiving portion is divided into a plurality of regions, canoutput luminance information of a subject for each region. There is nolimitation to the number of divisions. Note that amplifier circuits forpixel signals and peripheral circuits for signal processing are formedin the image sensor in addition to the pixels arranged in a lightreceiving portion.

A finder optical system is constituted by the pentaprism 4. Although notillustrated in FIG. 2 , a subject image reflected by the pentaprism 4can be observed through an eyepiece. A portion of off-optical axiscomponents of the light beam that has been reflected by the main mirror3 and diffused by the focusing screen 12 is incident on the AE sensor 6.The interchangeable lens 2 communicates with the camera body 1, asnecessary, through a contact of a lens mount provided in the camera body1. Note that, when a live view display is performed or a moving image isrecorded, since the main mirror 3 has jumped up, exposure control andfocus adjustment control are performed using information regarding acaptured image.

FIG. 3 is a block diagram illustrating an exemplary configuration ofelectric circuits of the camera body 1 and the interchangeable lens 2shown in FIG. 2 . A control unit 21 in the camera body 1 is a one-chipmicrocomputer that incorporates therein an ALU (ARITHMETIC and LogicUnit), a ROM, a RAM, an A/D converter, a timer, a serial communicationport (SPI), and the like, for example. The control unit 21 controlsoperations of the camera body 1 and the interchangeable lens 2 byloading a program stored in the ROM to the RAM and executing theprogram, for example. The specific operations of the control unit 21will be described later.

Output signals of the AF sensor 9 and the AE sensor 6 are input to anA/D converter input terminal of the control unit 21. A signal processingcircuit 25 controls the image sensor 10 in accordance with theinstruction of the control unit 21, and obtains an image signal byapplying A/D conversion and signal processing on a signal output fromthe image sensor 10. Also, the signal processing circuit 25 performsnecessary image processing such as compression and composition when anobtained image signal is to be recorded. A memory 28 is a DRAM or thelike, and is used as a work memory when the signal processing circuit 25performs various types of signal processing, or a VRAM when an image isdisplayed in a later-described display 27. The display 27 is either aliquid-crystal display provided on the rear side of the camera body 1 oran external display. The display 27 displays information such as settingvalues of the camera, a message, and a menu screen and a captured image.The display 27 is assumed to be an HDR-capable display. The display 27is controlled by an instruction from the control unit 21. A storage unit26 is a semiconductor memory card, for example, and receives a capturedimage signal input from the signal processing circuit 25.

A motor 22 follows the control of the control unit 21, controls up/downmovement of the main mirror 3 and the first reflecting mirror 7, andperforms charging of the mechanical shutter 11. A console unit 23 is aninput device group such as a switch that is used when a user operatesthe camera. The console unit 23 includes a release switch forinstructing to start a shooting preparation operation and to startshooting, a shooting mode selection switch for selecting the shootingmode, an arrow key, a determination key, and the like. A contact unit 29has contacts for performing communication with the interchangeable lens2. Input/output signal lines of a serial communication port of thecontrol unit 21 are connected to the contacts. A shutter drive unit 24,which is connected to an output terminal of the control unit 21, drivesthe mechanical shutter 11.

A contact unit 50 that forms a pair with the contact unit 29 is providedin the interchangeable lens 2. A lens control unit 51, which is aone-chip microcomputer similar to the control unit 21, is connected tothe contact unit 50, and can communicate with the control unit 21. Thelens control unit 51 includes a microprocessor, a ROM, and a RAM, forexample, and controls operations of the interchangeable lens 2 based onthe instruction from the control unit 21 by loading a program stored inthe ROM to the RAM and executing the program. Also, the lens controlunit 51 notifies the control unit 21 of information such as a status ofthe interchangeable lens 2. A focus lens drive unit 52 is connected toan output terminal of the lens control unit 51, and drives a focus lens.A zoom drive unit 53 changes the angle of view of the interchangeablelens in accordance with the control of the lens control unit 51. Anaperture drive unit 54 adjusts the aperture amount of the aperture inaccordance with the control of the lens control unit 51.

When the interchangeable lens 2 is attached to the camera body 1, datacan be communicated between the lens control unit 51 and the controlunit 21 of the camera body via the contact units 29 and 50. Also,electric power for driving the motor and actuator in the interchangeablelens 2 is supplied through the contact units 29 and 50. Information suchas optical information unique to the lens that is used by the controlunit 21 of the camera body to perform focus detection and an exposurecomputation, and information regarding the subject distance based on adistance encoder are output from the lens to the control unit 21 of thecamera body through data communication. Also, focus adjustmentinformation and aperture information that have been obtained as a resultof the control unit 21 of the camera body having performed focusdetection and an exposure computation are output from the control unit21 of the camera body to the lens through data communication, and thelens controls the focus lens in accordance with the focus adjustmentinformation, and controls the aperture in accordance with the apertureinformation.

Hereinafter, specific operations from shooting to development in thefirst embodiment will be described using FIG. 1 and FIGS. 3 to 8 . Whenthe control unit 21 enters an operable state by the power switchincluded in the console unit 23 in FIG. 3 having been turned on or thelike, first, the control unit 21 performs communication with the lenscontrol unit 51 of the interchangeable lens 2, and performsinitialization processing such as obtaining various types of lensinformation necessary for focus detection and photometry. Also, varioussettings made by a user are received through the console unit 23, andany shooting mode is set. When a shutter switch included in the consoleunit 23 is pressed half-way, the control unit 21 starts operations of AF(autofocus) processing and AE (automatic exposure) processing.Thereafter, when the shutter switch is pressed fully, the control unit21 starts operations of shooting.

Thereafter, development of a RAW image obtained by shooting isperformed. The development of a RAW image will be described using FIG. 4. In FIG. 4 , the reference signs 402 to 405 and 408 to 412 denotefunctional blocks, each of which indicates processing performed by thecontrol unit 21, but one or more of the functional blocks may berealized as a hardware circuit. Each piece of pixel data thatconstitutes RAW image data 401 indicates the intensity in the color of acolor filter provided in a corresponding pixel, and does not includeinformation regarding other colors. Here, it is assumed that one ofcolor filters of red (R), green (G), and blue (B) is provided in eachpixel of the image sensor.

A white balance unit 402 performs processing for correcting colorfogging due to the light source in order to reproduce white.Specifically, the white balance unit 402 plots respective pieces of RGBdata of pixels constituting the RAW image data 401 in a predeterminedcolor space such as an xy color space. Also, the white balance unit 402integrates R, G, and B values of data that is plotted in the vicinity ofa point, on the blackbody radiation locus, that is highly possibly thelight source color in the color space, and obtains white balancecoefficients G/R and G/B of the R and B components from the integratedvalue. The white balance unit 402 performs white balance processing byapplying the white balance coefficients to image data.

A color interpolation unit 403 generates a color image including allpieces of R, G, and B color information with respect to all of thepixels by performing noise reduction processing and color interpolationprocessing. A basic color image is generated by subjecting the generatedcolor image to processing performed by a matrix conversion unit 404 andgamma conversion unit 405. Here, the gamma characteristic in the case ofHDR development in the gamma conversion unit 405 is an inversecharacteristic of EOTF (Inverse EOTF), and is an inverse characteristicof PQ (Perceptual Quantization) (FIG. 6 ) shown in FIG. 6 , for example.Note that OETF (Optical-Electro Transfer Function) in which Inverse EOTFand OOTF (Opto-Optical Transfer Function) characteristics are combinedmay be used.

The gamma conversion unit 405 performs gamma conversion using a contrastadjustment curve (also referred to as a gamma characteristic, a tonecharacteristic for contrast adjustment, or a gamma curve). Shootingmodes in which output dynamic ranges are different as shown in FIG. 1are installed in a known camera, and a user selects one of the modes tobe used for shooting depending on the scene. Here, the difference in thedynamic range between two modes causes a difference in the peakluminance when HDR development is performed. Similarly, the dynamicrange may differ depending on the ISO speed. In the image sensor,relative to the reference ISO speed 100 in which amplifier gain is zero,the dynamic range increases at higher sensitivity at ISO 200 or higherat which an analog gain or a digital gain is applied, because the chargeamount is smaller relative to the capacity of the photodiode. Also, withrespect to the intermediate ISO speed in units of one third stop, in animage sensor including a low cost amplifier whose gain step correspondsto only one stop, the intermediate ISO speed is realized by gain-up orgain-down from the representative ISO speed.

Here, when gain-down is performed, because the saturated signal leveldecreases, the dynamic range decreases. In this way, there are caseswhere the peak luminance differs depending on shooting settings,specifically depending on the ISO speed. Therefore, the gamma curveswhose number corresponds to the number of combinations between theshooting modes and the ISO speeds are pre-stored in the memory 28 ascontrast adjustment curves 407. Also, a gamma selection unit 408 refersto the shooting settings (shooting mode and ISO speed) 406, reads outthe contrast adjustment curve 407 corresponding to the combination ofthe shooting mode and the ISO speed from the memory 28, and outputs thecontrast adjustment curve 407 to an output peak luminance informationacquisition unit 409.

Next, in step S501 (FIG. 5 ), the output peak luminance informationacquisition unit 409 acquires absolute luminance corresponding to eachoutput value of the gamma curve from the acquired contrast adjustmentcurve 407. The absolute luminance value can be acquired using EOTF suchas PQ (FIG. 6 ) or HLG. Also, the output peak luminance informationacquisition unit 409 acquires the maximum value of output values of thecontrast adjustment curve 407 or the nits value corresponding thereto.Note that, in this specification, the maximum value of output values ofthe gamma curve data, that is, the tone value at the peak luminancevalue in the output dynamic range, or the nits value correspondingthereto is referred to as maxDRL (maximum Dynamic Range Level).

Thereafter, a color luminance adjustment unit 410 performs processingfor improving the visual quality of an image on a color image, and imagecorrection in which an evening scene is detected and chroma enhancementis performed, for example, is performed according to the scene. Uponcompleting the color luminance adjustment, a compression unit 411compresses a high resolution image using a method such as HEVC. Also, instep S502, a recording unit 412 generates an HDR signal 413 bydescribing maxDRL acquired by the output peak luminance informationacquisition unit 409 in metadata of a file, and outputs the file in stepS503. Accordingly, the maxDRL is recorded in association with the HDRsignal 413.

FIG. 7 is an example of a file structure for recording an HDR signal.Header information such as a thumbnail image 72 is stored in an areaindicated by the reference sign 71, and information regarding an actualimage such as metadata 74 and actual image data 75 is stored in an areaindicated by the reference sign 73. In step S502, maxDRL may be recordedin the metadata 74 in FIG. 7 or in another specific area. Note that,because the depth of a known JPEG format is 8 bits, although the depthof at least 10 bits is needed in order to express an HDR signal in PQ,in actuality, a container for an HDR still image needs to be newlyadopted. Here, a container in HEIF (High Efficiency Image File Format),which is an image file format defined by MPEG (Moving Picture ExpertsGroup)-H Part 12 (ISO/IEC 23008-12), is adopted. HEIF has a feature thatnot only an actual image, but a thumbnail, a plurality of temporallyrelated files, and metadata such as Exif and XMP can be stored in onefile. Therefore, a 10-bit image sequence encoded in HEVC can also bestored in the file. The aforementioned maxCLL is not defined in the HEIFstandard, but maxDRL in addition to maxCLL is also stored in MakerNoteof Exif, in this proposal.

Moreover, the method of performing tone mapping for mapping the HDRsignal 413 to an SDR signal using maxDRL in order to view the HDR signal413 in the SDR environment will be described using FIG. 8 . Thisprocessing can be performed by the control unit 21 or the signalprocessing circuit 25, and here, the processing is performed by thecontrol unit 21.

In step S801, the control unit 21 sets an upper limit value Lmax and alower limit value Lmin for displaying the HDR signal in a full range ofEOTF of the SDR signal. For example, Lmax can be set to 2,000 nits andLmin can be set to 100 nits, but there is no limitation thereto. Also,the upper limit value Lmax and the lower limit value Lmin may be set bya user.

Next, in step S802, the control unit 21 acquires maxDRL from metadata ofthe file in which the HDR signal is recorded.

In step S803, the control unit 21 compares Lmax with maxDRL. Here, ifmaxDRL is greater than or equal to Lmax, the control unit 21 performstone mapping such that Lmax is the maximum value of the EOTF domain ofSDR, in step S804.

On the other hand, if maxDRL is less than Lmax, the control unit 21compares maxDRL with Lmin, in step S805. If maxDRL is less than or equalto Lmin, the control unit 21 performs tone mapping such that Lmin is themaximum value of the EOTF domain of SDR, in step S806. Also, if maxDRLis greater than Lmin, the control unit 21 performs tone mapping suchthat maxDRL is the maximum value of the EOTF domain of SDR, in stepS807.

Next, in step S808, the control unit 21 displays the SDR signal obtainedby tone mapping in the display 27.

In step S809, the control unit 21 inquires a user of whether or not theSDR signal subjected to tone mapping is saved in an SDR format, bydisplaying GUI in the display 27, for example. If the user instructsthat the SDR signal is to be saved through the console unit 23, forexample, the control unit 21 saves the SDR signal in the SDR format inthe storage unit 26, in step S810. If the user instructs that the SDRsignal is not to be saved, the control unit 21 ends the processingwithout saving the SDR signal.

As described above, in the present embodiment, when an HDR signal isrecorded in a file, maxDRL, which is peak luminance informationregarding the output dynamic range, is described as metadata. With this,an HDR signal can be mapped to an SDR signal considering the dynamicrange, and the difference in brightness can be suppressed between a casewhere the HDR signal is viewed in an HDR environment, and a case wherethe HDR signal is mapped to an SDR signal and viewed in an SDRenvironment.

Second Embodiment

Next, a second embodiment of the present invention will be describedusing FIGS. 9 to 11 . In the first embodiment, a method is described inwhich, when an HDR signal is recorded in a file, maxDRL, which is peakluminance information regarding the output dynamic range, is describedas metadata. In the second embodiment, a method of updating maxDRL whenretouching such as tone curve correction, in which the peak luminance inthe output dynamic range is to be changed, has been performed on an HDRsignal that has been recorded in a file will be described.

In FIG. 9 , the reference signs 902 to 905 denote functional blocks eachof which indicates processing performed by the control unit 21, but oneor more of the functional blocks may be realized as a hardware circuit.

In step S1001 (FIG. 10 ), an output peak luminance informationacquisition unit 902 acquires maxDRL described in metadata of an HDRsignal 901.

In step S1002, a color luminance adjustment unit 903 displays an imagebased on the HDR signal 901 and GUI for image retouching such as tonecurve correction in a display 27. Also, the color luminance adjustmentunit 903 corrects the color and luminance of an image in accordance withthe user instruction made through a console unit 23.

In step S1003, an output peak luminance information calculation unit904, after color luminance adjustment, calculates peak luminanceinformation maxDRL_r of the output dynamic range on which imagecorrection by retouching is reflected, with respect to maxDRL acquiredby the output peak luminance information acquisition unit 902. The tonevalue y after correction can be obtained using following Equation 1,where x is the tone value of an input signal before correction and f isthe input-output characteristic for correction.y=f(x)  Equation 1

Here, when the correction using the input-output characteristic f isperformed on tone values x in a range of 0≤x≤maxDRL, and ymax is themaximum value of tone values y after correction, the tone values y ofthe HDR signal after correction satisfies 0≤y≤ymax, due to tone curvecorrection. Here, the output peak luminance information calculation unit904 obtains maxDRL_r using following Equation 2.maxDRL_r=ymax  Equation 2

An example of retouching and an exemplary calculation of maxDRL_r areshown in FIGS. 11A to 11C. FIG. 11A shows a luminance histogram beforeretouching, FIG. 11B shows a luminance histogram when retouching isperformed so as to increase the luminance, and FIG. 11C shows aluminance histogram when retouching is performed so as to decrease theluminance. When retouching as shown in FIG. 11B is performed, maxDRL_ris larger than maxDRL, and when, conversely, retouching as shown in FIG.11C is performed, maxDRL_r is smaller than maxDRL.

In step S1004, the recording unit 905 records maxDRL_r to metadata 74 ofthe HDR signal.

In step S1005, the recording unit 905 records the file of the HDR signalto the storage unit 26.

Also, as a result of applying the first embodiment to a retouched HDRsignal and performing tone mapping such that maxDRL_r is the maximumvalue of the EOTF domain of SDR, the retouched HDR signal can be viewedat appropriate brightness in the SDR environment.

Third Embodiment

Next, a third embodiment of the present invention will be describedusing FIG. 12 . In the first and second embodiments, a procedure forviewing a saved HDR signal in the SDR environment has been described. Inthe present embodiment, a method of performing tone mapping in realtime, when retouching such as tone curve correction is performed on anHDR signal in the SDR environment, in order to view the HDR signal onwhich retouching is being performed or retouching have been completed,in the SDR environment, will be described.

Output dynamic range peak luminance information maxDRL_r to whichretouch information is added is calculated, in steps S1201 to S1203,using a procedure similar to that in step S1001 to S1003 in the secondembodiment. Next, in steps S1204 to S1210, the HDR signal is tone-mappedto an SDR signal with a procedure similar to that in steps S801 to S808in the first embodiment, and displayed in the display 27. The displayhere corresponds to viewing in the SDR environment.

Next, in step S1211, the control unit 21 inquires a user of whether ornot the retouched HDR signal is saved as a new file, by displaying GUIin the display 27, for example. If the user instructs that the HDRsignal is to be saved through the console unit 23, for example, thecontrol unit 21 saves the HDR signal in which maxDRL_r is saved in themetadata 74 into the storage unit 26 in step S1212, and advances theprocessing to step S1213. Also, if the user instructs that the HDRsignal is not to be saved, the control unit 21 advances the processingto step S1213 without saving the HDR signal.

Since the processing in steps S1213 to S1214 is similar to theprocessing in steps S809 to S810 in the first embodiment, thedescription thereof will be omitted.

According to the present embodiment, even if an HDR signal is retouchedin the SDR environment, maxDRL is saved in a file of the retouched HDRsignal, and the file can be recorded. Therefore, when an HDR signal isviewed in the SDR environment, as a result of maxDRL being referred to,the maximum luminance value of the output dynamic range after retouchingcan be mapped to the maximum value of the EOTF domain of the SDR signal.

Note that, when a RAW file is developed to a file in an HDR format usingdevelopment software or the like in the SDR environment as well, an HDRsignal that has been tone-mapped to an SDR signal using the procedure ofthe third embodiment may be displayed.

Fourth Embodiment

Next, a fourth embodiment of the present invention will be described. Inthe second embodiment, when the peak luminance is changed due to animage retouch, maxDRL as metadata is updated. On the other hand, in thepresent embodiment, even in a case where the peak luminance is changeddue to an image retouch, maxDRL as metadata is not updated. A case isassumed where the peak luminance of an image is changed by retouching inwhich sharpness strength is changed, for example, and maxDRL is updated.In this case, if tone mapping to SDR is performed based on maxDRL, theimage tone is changed due to the sharpness strength. However, when auser desires to confirm an image while changing the sharpness strength,for example, it is desirable that only the effect of sharpness is to bechanged without changing the image tone. Therefore, in the presentembodiment, even in a case where the peak luminance is changed due to animage retouch, maxDRL as metadata is not updated. Note that aconfiguration may be adopted in which one of a mode in which maxDRL isupdated as in the second embodiment and a mode in which maxDRL is notupdated as in the present embodiment can be selected by user operationor automatically.

Fifth Embodiment

In the embodiments described above, descriptions have been given mainlyassuming a still image file. In the present embodiment, an embodiment inwhich a moving image file is assumed will be described.

FIG. 13A shows a data structure of an MP4 format, which is an example ofa moving image file format. The data structure of the MP4 formatincludes an area 1301 for recording metadata, and an area 1304 forrecording a stream 1305 of a moving image. The area 1301 of these areasincludes an Exif area 1302 and a thumbnail area 1303, and maxDRL isrecorded in the Exif area 1302. However, there are cases where the peakluminance, in moving image data, changes in the middle of the movingimage data that has been continuously recorded, when the mode waschanged while recording was performed, for example. In this case, whatis to be recorded as maxDRL in the Exif area 1302 is an issue.Therefore, in the present embodiment, the peak luminance based onsettings when shooting of the moving image is started (peak luminance ina start frame of moving image data) is recorded in the Exif area 1302 asmaxDRL. Note that another value may be used as maxDRL, as long as thevalue represents the peak luminance of the moving image data. Forexample, peak luminance in an end frame of the moving image data, amaximum value of the peak luminance in all the frames, a minimum valueof the peak luminance in all the frames, an average value of the peakluminance in all the frames, a median value of the peak luminance in allthe frames, or the like can be used.

Also, there is a file format in which metadata can be recorded in eachframe, as shown in FIG. 13B. Areas 1311, 1312, 1313, and 1314 aresimilar to the areas 1301, 1302, 1303, and 1304 in FIG. 13A. An area1315 is a first frame of a moving image, and an area 1317 is a secondframe of the moving image. Also, the file format in FIG. 13B includes ametadata area for each frame. An area 1316 is a metadata area for thefirst frame, and an area 1318 is a metadata area for the second frame.When such a file format is adopted, the peak luminance in each frame maybe recorded as maxDRL. In this case, the peak luminance in each frame isrecorded in each of the metadata areas 1316 and 1318 for the respectiveframes. Of course, the representative peak luminance described above mayfurther be recorded in the Exif area 1312, which is a metadata area forthe entire file.

Other Embodiments

In the embodiments described above, maxDRL is described as the metadataof a file, but the mode is not limited to the mode in which maxDRL ishandled as metadata of a file. For example, the following mode isconceivable when a digital camera and an HDR-capable display aredirectly connected through HDMI, and an HDR signal is output. That is, amode is conceivable in which the digital camera outputs a valuecorresponding to maxDRL, to a recording medium for outputting signals,as metadata of the HDR signal, and this value is input to theHDR-capable display.

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2018-164801, filed on Sep. 3, 2018, and Japanese Patent Application No.2019-141685, filed on Jul. 31, 2019, which are hereby incorporated byreference herein in their entirety.

What is claimed is:
 1. An image processing apparatus that records animage signal as a file, comprising: one or more processors that executeinstructions stored in a memory to function as: an acquisition unitconfigured to acquire information indicating peak luminance of oneoutput dynamic range, of a plurality of different output dynamic rangesthat are set in the image processing apparatus in advance, in accordancewith shooting settings of the image signal; and a recording unitconfigured to record a first value based on the information acquired bythe acquisition unit to the file along with the image signal.
 2. Theimage processing apparatus according to claim 1, further comprising animage sensor for capturing an image of a subject, wherein the imagesignal is generated from a signal obtained from the image sensor, theshooting settings include a shooting mode, and the shooting mode is oneof a plurality of shooting modes corresponding to the plurality ofdifferent output dynamic ranges.
 3. The image processing apparatusaccording to claim 2, wherein tone characteristics for contrastadjustment that are applied to a signal obtained from the image sensordiffer according to a shooting mode included in the shooting settings.4. The image processing apparatus according to claim 3, wherein theplurality of different output dynamic ranges are determined inaccordance with the tone characteristics for contrast adjustment, andthe peak luminance is a maximum output value in the tone characteristicsfor the contrast adjustment of the one output dynamic range.
 5. Theimage processing apparatus according to claim 1, wherein the one or moreprocessors execute instructions stored in the memory to further functionas: an update unit configured to update the first value when the peakluminance has changed due to correction of color or luminance performedon the image signal.
 6. The image processing apparatus according toclaim 1, wherein the first value is not updated even when the peakluminance has changed due to correction of color or luminance performedon the image signal.
 7. The image processing apparatus according toclaim 1, wherein the one or more processors execute instructions storedin the memory to further function as: a conversion unit configured toconvert the image signal to a signal in a predetermined format byconverting tone values of the image signal to tone values in thepredetermined format having a different dynamic range, based on thefirst value.
 8. The image processing apparatus according to claim 7,wherein the different dynamic range is a standard dynamic range (SDR).9. The image processing apparatus according to claim 1, wherein the fileis recorded in a high efficiency image file format (HEIF).
 10. The imageprocessing apparatus according to claim 1, wherein the image signal is ahigh dynamic range (HDR) signal.
 11. The image processing apparatusaccording to claim 1, wherein signal characteristics of the image signalconforms to the Perceptual Quantization (PQ) standard.
 12. The imageprocessing apparatus according to claim 1, wherein signalcharacteristics of the image signal conforms to a standard that handlesdisplay luminance as an absolute value.
 13. The image processingapparatus according to claim 1, wherein: the acquisition unit furtheracquires second information that indicates a maximum luminance of theimage signal; and the recording unit further records a secondinformation to the file.
 14. The image processing apparatus according toclaim 13, wherein the peak luminance of the one output dynamic range isdifferent from the maximum luminance of the image signal.
 15. The imageprocessing apparatus according to claim 13, wherein the peak luminanceof the one output dynamic range varies depending on an ISO spped speedused to shoot the image signal.
 16. An image processing apparatus thatprocesses an image signal, comprising: one or more processors thatexecute instructions stored in a memory to function as: an acquisitionunit configured to acquire a first value indicating peak luminance,wherein the first value is recorded in association with the image signaland is information indicating peak luminance of one output dynamicrange, of a plurality of different output dynamic ranges that are set inadvance, in accordance with shooting settings of the image signal, and aconversion unit configured to convert the image signal to a signal in apredetermined format by performing a tone-mapping that maps tone valuesof the image signal to tone values in the predetermined format having adifferent dynamic range, based on the first value.
 17. The imageprocessing apparatus according to claim 16, wherein the conversion unit,when the first value is smaller than a predetermined second value,performs a tone-mapping in which the first value is mapped to a maximumtone value in the predetermined format.
 18. The image processingapparatus according to claim 16, wherein the conversion unit, when thefirst value is larger than the predetermined second value, performs atone-mapping in which the second value is mapped to a maximum tone valuein the predetermined format.
 19. The image processing apparatusaccording to claim 16, wherein the image signal is a high dynamic range(HDR) signal and a signal in the predetermined format is a standarddynamic range (SDR) signal.
 20. The image processing apparatus accordingto claim 16, wherein the image signal is recorded as a file in a highefficiency image file format (HEIF).
 21. The image processing apparatusaccording to claim 16, wherein signal characteristics of the imagesignal conforms to the Perceptual Quantization (PQ) standard.
 22. Theimage processing apparatus according to claim 16, wherein signalcharacteristics of the image signal conforms to a standard that handlesdisplay luminance as an absolute value.
 23. The image processingapparatus according to claim 16, wherein a value that indicates amaximum luminance of the image signal and is different from the firstvalue, is associated with the image signal.
 24. A control method of animage processing apparatus that records an image signal as a file, thecontrol method comprising: acquiring information indicating peakluminance of one output dynamic range, of a plurality of differentoutput dynamic ranges that are set in the image processing apparatus inadvance, in accordance with shooting settings of the image signal; andrecording a first value based on the information acquired by theacquiring to the file along with the image signal.
 25. A control methodof an image processing apparatus that processes an image signal, thecontrol method comprising: acquiring a first value indicating peakluminance, wherein the first value is recorded in association with theimage signal and is information indicating peak luminance of one outputdynamic range, of a plurality of different output dynamic ranges thatare set in advance, in accordance with shooting settings of the imagesignal, and converting the image signal to a signal in a predeterminedformat by performing a tone-mapping that maps tone values of the imagesignal to tone values in the predetermined format having a differentdynamic range, based on the first value.
 26. A non-transitorycomputer-readable medium that stores instructions executable by acomputer, wherein the instructions, when executed by the computer,causes the computer to operate as an image processing apparatus thatrecords an image signal as a file, comprising: an acquisition unitconfigured to acquire information indicating peak luminance of oneoutput dynamic range, of a plurality of different output dynamic rangesthat are set in the image processing apparatus in advance, in accordancewith shooting settings of the image signal; and a recording unitconfigured to record a first value based on the information acquired bythe acquisition unit to the file along with the image signal.
 27. Anon-transitory computer-readable medium that stores instructionsexecutable by a computer, wherein the instructions, when executed by thecomputer, causes the computer to operate as an image processingapparatus that processes an image signal, comprising: an acquisitionunit configured to acquire a first value indicating peak luminance,wherein the first value is recorded in association with the image signaland is information indicating peak luminance of one output dynamicrange, of a plurality of different output dynamic ranges that are set inadvance, in accordance with shooting settings of the image signal, and aconversion unit configured to convert the image signal to a signal in apredetermined format by performing a tone-mapping that maps tone valuesof the image signal to tone values in the predetermined format having adifferent dynamic range, based on the first value.